ReBNN: Resilient Binary Neural Network

87

in which γⁿ

i

is a balanced parameter. Based on the objective, the weight gradient in

Eq. (3.141) becomes:

δwⁿ

i ^{= ∂L}

∂wⁿ

i

+ γⁿ

i ^(wn

i ^−αn

i ^bwn

i )

= αⁿ

i ^{( ∂L}

∂ˆwⁿ

i

⊛1|wⁿ

i ^{|≤1 −γn}

i ^bwn

i ) + γⁿ

i ^wn

i ^.

(3.144)

The Sⁿ

i ^(αn

i ^{, wn}

i ^{) = γn}

i ^(wn

i ^−αn

i ^bwn

i ) is an additional term added in the backpropagation

process. We add this element because too small αⁿ

i ^{diminishes the gradient δwn}

i ^{and causes}

a constant weight wⁿ

i ^{. In what follows, we state and prove the proposition that δwn}

i,j ^is

a resilient gradient for a single weight wⁿ

i,j^{. Sometimes we omit the subscript i, j and the}

superscript n for an easy representation.

Proposition 1. The additional term S(α, w) = γ(w −αb^w) achieves a resilient training

process by suppressing frequent weight oscillation. Its balanced factor γ can be considered

the parameter that controls the appearance of the weight oscillation.

Proof: We prove the proposition by contradiction. For a single weight w centering around

zero, the straight-through-estimator 1|w|≤1 = 1. Thus, we omit it in the following. Based

on Eq. (3.144), with a learning rate η, the weight updating process is formulated as:

w^t+1 = w^t −ηδwt

= w^t −η[α^t( ^∂L

∂ˆw^{t −γbwt) + γwt]}

= (1 −ηγ)w^t −ηα^t( ^∂L

∂ˆw^{t −γbwt)}

= (1 −ηγ)

w^t −

ηα^t

(1 −ηγ)^{( ∂L}

∂ˆw^{t −γbwt)}

,

(3.145)

where t denotes the t-th training iteration and η represents the learning rate. Different

weights share different distances from the quantization level ±1. Therefore, their gradients

should be modified according to their scaling factors and current learning rate. We first

assume the initial state b^wt = −1, and the analysis process applies to the case of initial

state b^wt = 1. The oscillation probability from iteration t to t + 1 is the following:

P(b^wt ̸= b^wt+1)



b^wt=−1 ^{≤P( ∂L}

∂ˆw^{t ≤−γ).}

(3.146)

Similarly, the oscillation probability from iteration t + 1 to t + 2 is as follows:

P(b^wt+1 ̸= b^wt+2)



b^wt+1=1 ^≤P(

∂L

∂ˆw^{t+1 ≥γ).}

(3.147)

Thus, the sequential oscillation probability from iteration t to t + 2 is as follows:

P((b^wt+1 ̸= b^wt+2) ∩(b^wt+1 ̸= b^wt+2))|bwt=−1

≤P



( ^∂L

∂ˆw^{t ≤−γ) ∩(}

∂L

∂ˆw^{t+1 ≥γ)}



,

(3.148)

which denotes that the weight oscillation occurs only if the magnitudes of

∂L

∂ˆw^{t and}

∂L

∂ˆw^t+1

are more significant than γ. As a result, its attached factor γ can be considered a

parameter used to control the occurrence of the weight oscillation.